UMNO 


3 1761 11632062 3 


P f a $-; ‘ /f - 7 | 
y 4 A Led” LAM / ca be Ak Ly i KL c 


CALL NO. 
CAI ; . 
BS 112 DOMINION BUREAU OF STATISTICS 
Census Division C. 
-67P07 | OTTAWA P, "erament 
ations 


. of + ie a Publication No. 7 


are a 
eS aed NS TLS ATI } 


ae 


THE TESTING PROGRAMME FOR THE 1971 CENSUS IN CANADA | 


by 


I.P. Fellegi and K.J. Krotki 


inted from Proceedings of the Social Statistics Section, 1967, 


ashington, D.C.: American Statistical Association, pp. 29-38 


Sa eae : 
= See ea 
SSSA tere eee 


/ \ ‘ ; 
yA ws0.4. as y 
4 otintin Lor LOL « 


? DOMINION BUREAU OF STATISTICS 
ny Census Division Gove 
2S 2 " OTTAWA eT pee, 


Ca tlons 


9 er as Publication No. 7 


Hd} 


} 


THE TESTING PROGRAMME FOR THE 1971 CENSUS IN CANADA 


by 


I.P. Fellegi and K.J. Krotki 


Reprinted from Proceedings of the Social Statistics Section, 1967, 


Washington, D.C.: American Statistical Association, pp. 29-38 


e 


aS 


- 
4 


THE TESTING PROGRAMME FOR THE 1971 CENSUS IN CANADA 


I.P. Fellegi and K.J. Krdétki 
eDominion Bureau of Statistics, Canada 


1. INTRODUCTION 

Data on Canadian population have four some- 
what unique, or at least rare, characteristics. 
First, they are based on the longest series of 
modern censuses stretching back just over 300 
years.l/ Of uneven territorial coverage, of 
uneven subject-matter content, without legisla- 
tive basis until 1841, the censuses were never- 
theless being taken with remarkable frequency and 
make up to date an impressive list of 441 enu- 
merations (7, p. vii). Secondly, the data refer 
to a population which among all the Westerm or 
developed populations had for most, probably all, 
of the historical period the highest fertility 
and consequently the highest proportions of chil- 
dren (21, p. vii). Thirdly, Canada conducted in 
1956 and in 1966, between the main decennial 
censuses, national enumerations, even if with a 
rather limited subject-matter content, 2/ 
Finally, all this wealth of demographic material 
has been used and analyzed to an extent which 
until a few years ago could be described only as 
modest, though the recent quickening of interest 
in problems of demography and social survey 
methodology in federal government departments, 
provincial governments, universities and research 
institutions, will soon, no doubt, rill this gap. 


The testing programme for the next censuses 
in Canada should be viewed as part of this gener- 
al development. Many previous censuses had their 
own test operations, particularly when departures 
from earlier content or earlier procedures were 
considered (e g. 8, p. 17). Th:+se tests were 
held typically some 18 months %efore the census 
day. On occasions they could :nvolve numbers as 
high as 100,000 persons. They were largeiy in 
the nature of dress rehearsals after which only 
minor changes were possible, i.e. they were not 
investigational. In particular three innovations 
should be mentioned which required extensive 
testing: the predistribution of the Agriculture 
Questionnaire in 1951, the partial completion of 
which must have lessened the enumerative burden 
on the enumerators; the introduction of mark sen- 
Sing in 1951 with consequent machine reading and 
the postal check on a national scale in 1961 in 
urban areas which had appropriate mailing service 


Ih any case, these and other tests made 
their contribution to the gradual improvement of 
the census procedures and census results without, 
however, departing from the basic method of a 
door-to-door canvass and enumeration through 
interviews. An exception to this statement is 
the sampling form in 1961 which was dropped off 
by enumerators, completed with regard to the 
sensitive subject of income (it also included 
fertility and migration) by the respondent and 
then picked up by the enumerator. One of the 
most important improvements was due to the 
establishment of eight permanent Regional Offices 
after the last war to conduct the monthly Labour 


Force Survey. At the same time these offices 
became the focal points of the census-taking 
activity as well, resulting in an undoubtedly 
better selection, training and control of the 
census enumerators. Another major improvement 
was due to the gradual emergence of a more con- 
scious and better-oriented training programme. 


1.1 Motivation of experimentation 


The first serious effort to measure errors 
in a Canadian Census was made in 1956. Much more 
ambitious programmes were mounted in 1961 and 
1966 and more realistic estimates of coverage and 
content errors were obtained. Projects designed 
to evaluate the 1966 Census are the subject of 
the second paper of this session. Hence, they 
will not be discussed in the present paper. It 
should be noted, however, that the evaluation 
programmes of the 1956 and 1961 Censuses had a 
considerable impact on our thinking (9, 10, 13, 
18). We were surprised to find that our censuses, 
after all the hard work that went into improving 
them, failed to enumerate 2.5-3.0 per cent of the 
population and that, in fact, the per cent under- 
enumeration went as high as 10 per cent for a few 
critical age-sex groups. We were surprised to 
find that the response variance of the census 
statistics on such questions as education, labour 
force status, industry and occupation was about 
as high as the sampling variance would have been 
with a 25 per cent sample (13). In addition, of 
course, these statistics were also subject to 
response biases as well. By far the largest por- 
tion of this response variance was accounted for 
by the so-called correlated response variance (16), 
i.e. roughly speaking the component due to the 
effect of an enumerator interviewing a substan- 
tial number of households (about 150 households 
were enumerated by one enumerator). In the 1961 
Census we have used sampling to a limited extent, 
but it is substantially true that most cf our 
census statistics had no sampling error but sub- 
Stantial response errors. Any cost-benefit analy- 
sis would indicate, that if we were able to reduce 
the response errors substantially at the price of 
introducing some controllable amount of sampling 
error, we would be better off. 


We need not emphasize to this audience thet 
the last ten years saw not only the development of 
some key experiments and important mathematical 
models leading to a better understanding of the 
limitations of our censuses, but it was at the 
same time a period of enormously increasing utili- 
zation of census statistics. Further, not only 
have our users become more numerous — they have 
become more sophisticated as well. Their needs 
are; more precise census statistics (even for 
relatively small areas and/or special sub-popula- 
tions), measuring more characteristics, available 
sooner, available in various different forms (in 
published table form, on tape, on punch cards, in 
the form of graphs), capable of being followed up 
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by special surveys and linkable with other data. 

£ was considered essential to ensure that the 
1971 Census incorporate a number of methodologi- 
cal changes to meet the requirements of census 
users during 1972-76. 


Two of the impulses leading to our pro- 
gramme of experimentation have been mentioned so 
far: efforts to improve understanding of the 
census-taking process and the needs of users. A 
third important impetus should be mentioned: the 
experiments carried out by the Bureau of the Cen- 
sus. These experiments demonstrated that: 


(a) it is feasible to establish an urban ad- 
dress register; 


(b) the Post Office can collaborate in material- 
ly improving such a register} 


(c) such a register can effectively be used for 
sampling purposes; 


(d) if a census form is mailed to addresses on 
the register, then a large proportion of 
householders will complete the question- 
naires in a machine-readable form, with 
relatively little follow-up; and 


(e) that such a mail census with follow-up 
holds out important possibilities for re- 
ducing the response errors and at the same 
time making some gains in reducing the 
coverage errors. 


The advantages and disadvantages of self- 
enumeration have been widely discussed (2, 33). 
We would like to emphasize only those of the ad- 
vantages which loomed high in our thinking: 


(i) the enumerators! contribution to the re- 
spouse variance sho.l: decrease sharply 
since the role of enurerators is re- 
stricted to follow-up; 


(ii) the early return of mailed question- 
naires to a central office 3/ permits an 
independent edit in time to trigger off 
an early follow-up, where necessary; 
this, we think might reduce both the re- 
sponse variance and the response bias}; 


(iii) each adult member of the household is 
able to answer the census questions for 
himself; 


(iv) respondents are able to consult records; 


(v) the publicity campaign can be made to 
"beak" during the mail-back period;4/ 


(vi) the cases of non-contact might be re- 
duced for people who are difficult to 
find at home but whose mail will reach 
them, a consideration particularly 
important in view of the increasing pro- 
portion of women participating in the 
labour force; 


(vii) coverage errors might be reduced, since 


each household would have several chances 
of being included in the census: during 
the preparation of the address register, 
during its improvements through subse- 
quent (mainly, post cffice) checks and 
through the intensive probing during the 
enumeration process; 


(viii) the address register, in machine-readable 
form, can facilitate geographic tabula- 
tions in table or graph form, it can faci- 
litate linkages of census data with data 
from other sources and it can facilitate 
the taking of speciai follow-up surveys. 


There was, clearly, a method of census- 
taking emerging, which held out important prom- 
ises, at least in urban areas.5/ Throughout the 
first half of 1966, discussions were held which 
led to the decision about midway through the year 
to conduct an experiment using this method. 


1.2 Some "boundary conditions" 


The problem of developing, testing and 
implementing a new method of census-taking had 
some restrictions and conditions attached to it. 
To borrow a mathematical phrase, these were the 
“boundary conditions" of our problem. 


The first unalterable condition was, natu- 
rally enough, that a census wili have to be taken 
on June 1, 1971. Working backwards from this 
date, it appeared that by the middle of 1969 the 
method of census-taking, the content of question- 
naires and all the important features of the field 
work will have to be "frozen". At this point a 
dress rehearsal will be held, but no major changes 
in procedures. The developmental and testing 
process will have to be carried out, therefore, 
during a two-year period. 


The second "boundary condition" was the 
limitation of staff. In mid-1966 when the green 
light was flashed for testing, there was no or- 
ganization or personnel available for full-time 
work on the test programme. The regular staff of 
the Census Division was working full steam on the 
processing and publication of the 1966 mid-decade 
census. A nucleus of full-time staff was bor- 
rowed, others had to squeeze in some part-time 
work. This staff situation in a very real sense 
determined the pace of our testing programme. The 
first test could not be scheduled for earlier than 
the fall of 1967, with a second series of tests in 
1968, and finally the dress rehearsal of 1969. 
This schedule put a very heavy burden on the 1967 
test. In fact, the 1968 tests will have to be 
planned largely, without the benefit of the re- 
sults of the 1967 test being eavailable.6/ 


1.3 Plan of the paper 


After these introductory remarks the remain- 
der of the subject will be discussed under four 
headings: an outline of and comments on the spe- 
cific method of census-taking which was tested in 
September 1967; an outline of the evaluation pro- 
gramme carried out in conjunction with this test; 
comments on alternative methods of producing an 
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address register; and a brief discussion of some 
of our future, plans. 


2. REASONS FOR AND OUTLINE OF THE LONDON TEST 


2.1 Reasons for the specific type of test 


The methodological changes which have to be 
tested before they could be relied upon include: 


(a) self-enumeration with or without use of 
mail; 


(b) use of address registers; 
(c) automatic geographic coding; 


(d) new computer hardware, including especially 
input devices; and 


(e) new computer software. 


Long discussions were held, and several position 
papers were written, covering a very large number 
of logically possible alternative methods of con- 
ducting and analyzing census tests. Without clos- 
ing our eyes entirely to other methods, alterna- 
tives compatible with the main aspects listed 
above were chosen on grounds of intuition, common 
sense and experience elsewhere, mainly in the 
U.S.A. The impact of any major methodological 
changes on operational procedures and subject- 
matter content also needed testing. Such impact 
would be felt on field edits, the flow and hand- 
ling of questionnaires, production of manuals, 
training and ocganization; all creating new pro- 
blems and requiring ner attention. Not all of 
them could, or indeed should, be tested in one 
1967 test. 


There was however no difficulty in selecting 
the obvious corner-stones for the structure of the 
testing programme. They were four in number: 


(i) construction of an address register; 


(ii) mailing-out and mailing-back of question- 
naires; 


(141i) self-enumeration on a questionnaire of 
sufficient length and complexity; and 


(iv) locally organized editing and follow-up 
procedures to deal with total non-response 
and partially (and/or inconsistently) com- 
pleted questionnaires. 


Naturally, these corner-stones would provide the 
opportunity for testing a host of other aspects, 
some of which have already been indicated briefly 
in paragraph 1.1 above, but it was understood that 
they would have to give way, if pressure of work 
and need for other attention, did not permit going 
outside these four main purposes. 

Clearly, such purposes excluded testing in 
remote areas with no mail delivery, in areas where 
the construction of address registers would be 
prohibitively expensive, and in areas with popu- 
lation centres so small that it would be uneco- 


nomic and unrealistic to organize a local office 
for the centralized edit and follow-up. 


The questionnaire eventually used was a 
householder questionnaire, rather neat and FOSDIC 
readable. It was of two kinds: short with only 
basic questions for every household and a long 
one for one in every four households with two 
facing pages per individual and well over 70 
questions, many with several sub-questions. 


2.2 The London Test 


The town of London in Ontario was selected 
out of eight candidates of comparable population 
size because of several characteristics which 
made it attractive for testing purposes. Its 
size (about 200,000 persons), its owner-tenant 
ratio (close to the national average), its con- 
siderable industry and occupation diversity, and 
its higher-than-average proportion of converted 
dwellings rendered it an attractive site for the 
test. Its low ethnic diversity made it unrepre- 
sentative, but it was decided to take on this 
problem at the next round of tests. 


In the event the preliminary and impres- 
sionistic opinion is that we have chosen "too 
well", The co-operation of the public and local 
authorities was of a high order. The publicity 
given and received was favourable and positive. 
There was an eagerness on the part of the public 
to help. Just over 85 per cent of questionnaires 
were (3) returned by mail, the great majority 
within a few days. This compares favourably with 
results obtained in similar tests in the U.S.A, 
(4, 31). The Telephone Answering Service gave 
assistance to almost 7 per cent of householders 
(3) which is higher than expected (6, 17). This 
again can be taken as an indication of the public 
eagerness to do ell. The selection of an "or- 
derly" city was intended to give the testa better 
than a fair chance to become an operational suc- 
cess. A very low response rate would not only 
knock out the cost-benefit basis for a mail ques- 
tionnaire, but would also probably knock out the 
whole idea altogether by showing that the Canadian 
public is not prepared to deal with a mail census. 


Within the proportion returned by mail the 
proportion acceptable without further field work 
is the next factor determining the economics of 
the new method. 


Within the proportion which fails edit 
specifications, the proportion of incomplete or 
inconsistent questionnaires which can be cleared 
through the telephone and which consequently re- 
quires no costly personal visit follow-up is the 
third important factor determining the economics 
of the new method. There are proportionately 
fewer telephones in Canada than in the U.S.A., 
but according to the telephone companies Cana- 
dians speak more and on the average, longer than 
natives south of the border. It cannot a priori 
be said whether the lower density will be made up 
by the apparently greater volubility or talka- 
tiveness and, therefore, ensure an economic 
follow-up by telephone. 


* 


+ 


aes both, Wiehe ane hey ses alee 2g 


, one Bt aa 7 ae at 


VT ni 3 


» vautbang widen ay 


yites thw eedete | 2shyipe a 
gual *., at. Aeayhen srs. eae WA ah 
ove) Fake ever bie ade yew | 

hs weve? Hew il sewbae ihe” iad ; 
wae itesup~ an dea one e die hides 


4 . 7% * 
Sudabeadiss . ef 


4 


ae a) 


LT y~ - 
finden tne) Kaw, oD BE. th ee oe ac: awe sith: 
aioe! # feo Abe Delhies Be abide grids to Ler 


saa a4 
si} vie 
nals ao 
Thane, 
She 


arr aasdiledebeacicen he 1 anrer x 
vii" sui ad cram ieta? 723° aries 
reiies awegh wah) Reet, OO aS = 
die) gh BOT ea! Bie) 
I et a ee 


| j 

tery Yen WAG? I'S aad “yave4hags ata yf o7) 
wa atcos ys * cp oo Seles ones sad 
eae tit 7 Wire beret he ei Gel era ieee 

Cg ode wo ete met Lhe la> PST Set, avilieinas 
4G) 4) celtaer Fel Bei igen eda oe ey Mors 

‘ - we oy vom iar me 4) t8 i ee a & ‘ 
weal esanelo Boeke ite #) ath ore re 
Tva'n$) chet, Wa hee : chptarhi fl fiw 
: Poh 039) os : ’ ‘er ‘ Wy, O01) Preity ys 

Micog iit per Wats hel? o veyorhae bend 

Te oy ht Fe syne ae HTS vas wri 
OF eb UNE VERE aS SHO Se iy SVD! eel cant od 
: thy Brie GAs ye s/t , Peeginrs ty (Se 
en mee Sater nif eve "te 44% 
ih Se each ae ce a LR EVES Pe Pea 
aid 


hh we FOS Sone AE Serge eens eS es 
¥ — 
Le tae Poe s i “Pl oan 


e: % y+ bee ts ps) Pe oS WE 'Hiy 5 
i SSA & Very By Fi ; val: «eS y ram 
ence peal a 5 vigetts: > oo Rporrenes 
y Siig ey t ye fe: Teed ooie Yeioet 
a : penny gy zt seat phak: & yee 
ine dios ait eS eS? aR ye his) We uray A” .aGRe 
emis dukaae gy AON & Seals eRe: Tet ge” 1 Oe ee 
/ pe sana S4ere Dekel Ai pthw: Ne tc not.) 
rel bei) AND Sit? 9? Wier! LetsrIopas)% av3, Sit 
igre sam 6 ist fase as Pigs cy ag ee oped Bf 

‘ ah : 
wo? Ldee ve? hs we : 4 ets” 7h 
uni “eh VsePP on [6¢. o4%¢ i Ss oRotd 
in Gadmotosy<s githe 1 Sct? wt 
ee ere syid* 
{3 Pl) a “aed ee OAT guiG wes. kde 
Ra 53.0) ce 14) WUTARSUGah ee” we i. )sem 
5pa4 5 $c. Wao. ot ew palin, 2 rye 7 a ener 
+  fvtre pe Nee Wop nWw hh ohh 1) off-beat 
Ah we Gare eps sete: J deeeAG, 19 on ween ep 
obebhBon (a0y oc lehrs (-REMEeida) Siitesarm Sot 
Viegbhulete wore sre 42aNT | SBolaee wea “rR 

P pre Es ‘Gai wy (rea * ce ‘ey nz para) sig yeh 


} seh ee Pre 
Y aise bre arte vielee manlt 
Tey oF Be Ge Wolpe Wey ian 
“ty ye VBA > 'y aeltt fal? iver a’ 
“wt oy iste » yim tite a's 
Yiskeieds (tuk qatar 
TRG Fe) indore 


- MRD BO COT ee AMete OE TOHF 3) hn 
i ae 
aSal tra 4% Wiohiee = ait 
alii, O59" 2 Pte eh 
‘> POPE? San Obl Pouio 
shAnperse HG qty) pee” 
4 4 7 2” =F 


ve 


even SUE 
af3. 06 


rhe rsodua ibatind 


, 4 
anu 


‘t dy awley par ee 
ssriatr oat af MAG BN. 
ye ay tet ‘ 
tahor, j te) he f 
‘* ood ml ey is 

usdeel Porcney, © Ph: 
TONGS. 4G rhe ae 
aay WS eae ars ie 3 
wo hd Bers Pe) oe ee 
<qet lime hae ines snare 
Sead Poe ML ay ‘geen 
“Sia Cin), WELT sae ee at i : 
« Ad suarine See aR Nn BSS Pi ythe 
Pe ates. eg mea TE Kee, Yate 
he heh SY lito head gan a 
ett Rl ali ae a huetetal an a). oto 
9 Aaa Ne 


a, a | eh 
<gapsliade ail ivenwe 

vsigh athe ad sioedh Psa 
yeugurr ct ane + Srp cent? oft 
Tete kpaty an sianeen Se octane 


’ van! 


<orkoaip Toston. ate © de FQen gt 
: 3 

“y See. 19 8 Ao} cia fat ms 

bre 322 aancte bere, ee arn 2 sae 


rent — <y 


aye ts 7, Ae on abt! tins J insane 
avmooegs-acm lasad Wiiw Thais cy sett 


Hoo (VL ease Lah Sti ae 


Cs oh} /etad, Biuitow 
al aera HiT ®. 2m 
fA niied fepep clad wae 
tat bes +o aa ae 
i") vitw 5 SF Unees 7% +e, b 


4 ew j derrer aon BA ha 


roe sie 
ov) at 


saree A pets 
ey5§C.1) sp 


It is not the purpose of this paper to 
dwell at any length on the complexity of the op- 
erations in the local office. To raise a large 
labour force at short notice for short periods, 
to train it in a large number of varied and com- 
plex tasks under supervisors who themselves are 
unfamiliar with the tasks (consider the setting 
of rates of pay as an example of the variegated 
problems), to design and work an organization 
which will process tens of thousands of pieces of 
paper, each with hundreds of entries, through 
many different steps, in many different places — 
these were all no mean tasks to be performed. 
The satisfaction that these unaccustomed labours 
can be carried out satisfactorily had to be se- 
cured. ; 


It goes without saying that however suc- 
cessful a test may be as an operation and however 
promising its economics, it must first of all 
give satisfaction on two points: there must be 
improved coverage and a higher quality of subject- 
matter content. 


2.3 Important differences between Canada and 
the U.S. 


In a number of ways Canada is similar to the 
U.S. This is a source of great help to us, since 
it enables us to learn effectively from the numer- 
Ous experiments conducted by the Bureau of the 
Census in the field of census-taking. We can 
avoid proven pitfalls and follow up the avenues 
that appear promising in the US experiments. Our 
colleagues in the Bureau of the Census have given 
us of their time and experience unstintingly, in- 
vited us as observers to their field trials and 
discussed with us with complete frankness both 
their successes and their failures. However, 
there are, in spite of the great similarities, 
important differences between Canada and the U.S. 
We shall list a few of the di*f:rences which are 
most significant from the poinc of view of census 
methods. 


Our census has to be bilingual. Every citi- 
zen is entitled to complete his questionnaire in 
either of the two official languages. This means 
that if we want to have a mail census, then in 
certain parts of the country we have to mail out 
in the same envelope both an English and a French 
questionnaire.7/ Doubling the amount of paper -to 
be addressed and mailed causes operational prob- 
lems but, more importantly, it may cause some 
response problems as well. The long form is long 
enough as it is and it may well frighten some po- 
tential respondents. Two long forms, even if one 
of them can be thrown away, may be the straw that 
breaks the camel's back. There may well be some 
problems in the office as well since question- 
naires have to be sorted by language for edit as 
well as telephone or field follow-up. 


A second problem relates to the Canadian 
winter and to the fact that there is no commercial 
mailing list. There is no commercial need for it 
since our Post Office is willing to distribute un- 
addressed advertising material to householders. 

We have to prepare therefore our own mailing list. 
This means that we have to have in the urban areas 
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two distinct field operations: first a listing 
job,8/ then the mail-out, mail-back census with 
field office edit and follow-up. One operation 
cannot smoothly blend into the other since after 
the listing of addresses and before the mail-out 
we need several months to key-punch these ad- 
dresses, prepare the appropriate workloads, imple 
ment one or two Post Office checks. For a census 
date of June 1 the ideal listing time would there- 
fore be some time in February or March. The Cana- 
dian winter being what it is, one tries to avoid 
extensive field work during these months. In 
fact, the last convenient opportunity is during 
October of the previous year. This has its dis- 
advantages, however, since our list likely becomes 
more outdated during those eight months than it 
would be during three or four months.9/ We have 
to rely on other sources, primarily the Post Of- 
fice to update our register, although some of the 
probing questions on the questionnaires themselves 
will hopefully improve coverage. 


A third important difference relates to 
scale which this time works primarily to our ad- 
vantage. The permanent field force of the Bureau 
of the Census has, as its prime function, the con- 
duct of the Current Population Survey. Similarly, 
the main task of our permanent field force is to 
take the Labour Force Survey. These two house- 
hold surveys have similar objectives and standards. 
Sampling being the undemocratic discipline that it 
is, we need about the same sample size as our 
American colleagues in spite of the fact that the 
Canadian population is only about one-tenth as 
large. This means, however, thet we have about 
ten times as high a sampling ratio, ten times as 
large an experienced field force per capita. This 
field force can do the bulk of the October listing 
job, minimizing the problem of hiring and training 
for this additional task that we have. Equally as 
important, we may well be able to rely on the per- 
manent field supervisory personnel to run the cen- 
sus field offices. 


A fourth important difference relates again 
to scale, but this time it is disadvantageous. 
Planning top management and anaiysis is very lit- 
tle related to the scale of operations. Conse- 
quently, the per capita investment on this type of 
activity has to be much higher in Canada than in 
the U.S.A. This difficulty, as indicated earlier, 
is somewhat relieved by the possibdility of drawing 
on the experiences of our professional colleagues 
elsewhere, particularly south of the border. 


3. EVALUATION PROGRAMME OF THE LONDON CENSUS TEST 


The evaluation of the London Test will, of 
course, be carried out on many fronts. Very im- 
portant aspects of it will be based on judgement 
and observation: how orderly was the operation, 
were instructions followed, was it possible to ad- 
here to the timetable of operations, could this 
timetable be condensed in some fashion, could such 
an operation be carried out on a national scale, 
etc. This type of evaluation provides the earliest 
assessment of the operation. Another key indi- 
cator, also available very early, is the response 
rate on short and long forms as well as the number 
of telephone and personal visit follow-ups on each. 
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The more formal evaluation of the test will 
be carried out under four headings. These are 
the evaluation of coverage, local office proce- 
dures, computer editing and contert. 


3.1 Coverage 


A person or household can get into the 
count under the London Test procedures through 
five streams: the field listing, the quality 
control of field listing operation and some of 
the relisting triggered by the quality control, 
the advance Post Office check, the final Post 
Office check and follow-up which may be triggered 
off by responses to certain probing questions on 
mailed questionnaires.10/ One of the objectives 
of the coverage evaluation programme is to esti- 
mate the additions and deletions of basic ad- 
dresses, subaddresses or households 11/ and per- 
sons from each of the five sources. Provision has 
been made to identify on the relevant records the 
source of additions and deletions. The total 
number of additions and deletions by source can 
also be tabulated by size of basic address, size 
of household, type of area (e.g., downtown ver- 
sus suburban), etc. 


An important aspect of evaluating the con- 
tribution to coverage of some of the operations 
is afforded by the quality control operations. 
The original field listing (15) was quality con- 
trolied as well as the advance Post Office check. 
The field listing was quality controlled through 
the relisting by supervisors of a sample of the 
more difficult blocks and the comparison of the 
supervisor's list with the lister's list (12). 
The advance Fost Office check was quality con- 
trolled by withholding a sample of addresses ori- 
ginally listed and by checking whether the Post 
Office made up a "missed address" card for 
them (1). 


The key measures from the coverage point of 
view will, of course, be the proportion of persons 
and households finally added by the combined pro- 
cedures listed above and estimates of proportions 
missed in spite of these procedures. The total 
number of households or persons added will be 
broken down into the number of households in add- 
ed basic addresses, as well as the number added 
in partially enumerated basic addresses. We shall 
also estimate the impact of definitional errors on 
the count of households. Similarly, the number of 
all persons in added basic addresses, added house- 
holds in partially enumerated basic addresses and 
partially enumerated households will be estimated 
separately. We shall also attempt to determine 
some of the characteristics of the added persons. 


Parallel to the measures of basic addresses, 
households and persons added through the five 
main streams, estimates of basic addresses, house- 
holds and persons missed in spite of these pro- 
cedures will be given. Some of these estimates 
will be based on the Post Office quality control, 
but the basic tool will be the Post-Enumeration 
Survey (PES). A brief description of the design 
of the PES might be in order.12/ 
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The blocks of London were stratified ac- 
cording to the likely deficiency of the address 
register. The information for stratification was 
obtained by comparing the address register with 
other lists and noting the number of addresses on 
the other lists which were not on our address 
register. A stratified sample of 20 per cent of 
the blocks was selected. Within the selected 
blocks every second basic address was selected to 
yield a 10 per cent sample of basic addresses, 


The first step of the Post-Enumeration 
Survey (coverage) was a very thorough field re- 
listing of the basic addresses within the select- 
ed blocks about two weeks before the census date. 
This relisting of basic addresses will form the 
basis of our estimates of the number of basic 
addresses missed by the census as well as of the 
number of households and persons missed in com- 
pletely missed basic addresses. 


The next step in the operation was that of 
the reenumeration. The.sample for the reenumera- 
tion consisted of the 10 per cent sample of basic 
addresses originally selected from the census 
address register plus all basic addresses which, 
on the basis of the relisting described above, 
appeared to have been missed by the census. Al- 
though from the point of view of sampling effi- 
ciency it would clearly have been advantageous, no 
subsampling was carried out within the basic ad- 
dresses, since we felt that subaddresses and 
households are not sufficiently unambiguous to be 
used as sampling units (an exception to this rule 
was made in large, regular apartment houses). In 
fact one of the objectives of the reenumeration is 
to learn something about the definitional prob- 
lems. The interviews and questionnaires were 
highly probing on coverage: both on coverage of 
persons within the households and on coverage of 
households at th: same basic address. A series 
of questions were asked probing the de facto pop- 
ulation as well.13/ 


Qn the content side the only questions asked 
of everyone in the PES sample were related to age, 
sex, marital status and relationship to the head 
of the household. Two housing questions of some 
coverage importance were also asked. A substan- 
tial amount of reconciliation was carried out to 
clear up discrepancies on both coverage and con- 
tent between the census and the reenumeration 
survey. In addition, a few weeks after the re- 
enumeration all persons who were apparently missed 
by the census were reenumerated once egain on the 
long questionnaire. The objective of this second 
reenumeration was to find out more about the char- 
acteristics of persons who were missed by the cen- 
sus. 


3.2 Local office procedures 


The following major activities are carried 
out in the local office or directed from the local 
office: mail check-in, edit of questionnaires, 
telephone follow-up, personal follow-up and coding. 
Ideally we would like to quality control ail of 
these operations, but at any rate at least we 
would like to evaluate them. 
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The check-in of the mail returns, since it 
was obviously an absolutely key operation, was 
100 per cent controlled, i.e. all mail returns 
were checked and all discrepancies-were recon- 
ciled. | 

The quality control of the edit operation 
took the form of acceptance sampling at the work 
unit level, i.e. all rejected work units were re- 
edited and the editors concerned were retrained. 
Only six editors out of 70 were affected by the 
quality control intervention and it is doubtful 
whether this operation was worthwhile in terms of 
its impact on quality. The speed of editing was 
too great to be caught by control. At best of 
times, through quality control operations we can 
control and estimate the average outgoing quality. 
However, since this operation was carried out in 
the field office in the heat of operations, the 
records of the quality control operation do not 
provide us with refined enough tools to evaluate 
in detail the edit operation and its impact on the 
final product. For purposes of evaluation, there- 
fore, a 10 per cent sample of the "short" part of 
the questionnaires was selected and reproduced 
(actually this was necessary anyway for the Post- 
Enumeration Survey as described above). These 
reproduced questionnaires were edited by the Head 
Office Staff after the close of the London Office. 
These specially edited questionnaires were then 
matched with the "short" pages of the original 
questionnaires which, by this time, were through 
the regular processing. A comparison of the ori- 
ginal questionnaires with the duplicate copies 
will enable us to evaluate in respect of the 
"short" pages the work of the various sections of 
the local office since the editors, the telephone 
follow-up enumerators and field follow-up enu- 
merators were all using pencils of different 
colour and their contributions can be distinguish- 
ed from each other. ; 


There is, at the time of writing this paper, 
no plan to evaluate the impact of editors on the 
long questionnaires and those housing parts of the 
short questionnaires which will be reproduced for 
PES purposes, though no doubt some manual study of 
their interventions will be evolved sooner or 
later. However, the study of the multi-coloured 
dots (the "'tallies') described in paragraph 3.4, 
although directed mainly towards assessing partial 
non-responders, may coincidentally provide for 
each, Or some questions estimates of the propor- 
tion of entries made by respondents, by editors, 
by the two kinds of follow-up and the proportion 
of entries left blank (the proportion of entries 
completed by respondents has an important impact 
on the response variance). It may thus provide 
estimates for each, or some questions on the 
Questionnaire of the effect of the various oper- 
ations in reducing the N.A. rate (i.e., the pro- 
portion of blanks which should not be blank). 


3.3 Computer editing 


No firm plans have been made for a formal 
evaluation of the set of computer programmes used 
for edit and imputation. We created some possi- 
bilities, however, which we intend to follow up 
and which might help us in assessing these pro- 


grammes. While the editing programmes are being 
readied, specifications for the comparison of the 
questionnaire tapes before and after are being 
worked out. The intention is to obtain a count 
of the interventions of the editing programme. 


The set of programmes will automatically 
and without manual intervention create a fully 
edited file and it will make all the imputations 
in the case of missing or inconsistent informa- 
tion. A summary will be obtained at the work 
unit level of all imputations made by the conm- 
puter. On the basis of these summaries the sub- 
ject-matter statisticians can decide to pass the 
work unit or to intervene manually. The program- 
mes will be assessed from several different 
points of view. The first point is: do they 
deliver as good a job as cierks? Other important 
questions are: in what ways should the program- 
mes be altered? How much imputation, at what 
area level, for what questions should subject- 
matter professionals let pass without review and 
at what point should they review the imputations? 
We think that the process itself of looking for 
answers to these questions will be very useful on 
account of its educational effect. 


A more formal evaluation of the effect of 
these programmes on the basic demographic var- 
iables of age, sex, marital status and relation- 
ship to head of household will take place through 
a matching of the Post-Enumeration Survey records 
with the edited census file. 


An interesting and very important problem 
relates to the computer assignment of family 
codes. A household, which typically wili report 
on one questionnaire, may contain more than cone 
family. The information on the questionnaire 
does not explicitly reflect all the possible com- 
plex family re'ations, only the relation of eacn 
member of the household to one person: the hear 
of the household. Nemes would provide a useful 
indication but names are not available in machine- 
readable form. They will be used by manual coders 
in coding families 14/ and then checked with the 
mechanical coding of families based on the few 
relevant pieces of information available for 
everyone: age, sex, maritai status and relation 
to head of household. 


3.4 Content evaluation 


A considerable number of tests of the con- 
tent of the questionnaire have been considered. 
They can be conveniently described by being 
grouped under three headings: those which are 
not likely to be undertaken (e.g., 5), those 
which will be undertaken with high priority in 
order to provide data for the decisions affecting 
the remaining parts of the testing programme, and 
those which will be done more thoroughly but which 
are unlikely to have an early impact on the future 
testing programme. We shall only outline the 
projects which will or are likely to be under- 
taken. 


There are several urgent, high priority 
projects. A quick tally will be made, question 
by question, cf blanks on a small sample of ques- 
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tionnaires (up to five were permitted by editing 
instructions on non-essential questions), of im- 
putations by manual editors, of completions by 
telephone follow-up, of completions by personal 
visit follow-up, of completions by non-response 
follow-up. The propertions under each question 
will be combined with cbservers!' impressionistic 
reports.. This will be the immediate basis on 
which questions will be redrafted, rearranged 
and relevant parts of the questionnaire design 
changed. A larger sample of questionnaires will 
be analyzed more intensively. Summaries from the 
editing at the microfilming stage may throw some 
modest light on some content issues. Question- 
naires obtained in the monthly Labour Force 
Survey in London will be matched, item by item, 
with the questionnaires obtained from the same 
households in the Census Test (about 400 house- 
holds are affected of whom 100 were enumerated on 
a@ long form in the Census Test). As indicated 
earlier the coverage PES questionnaire has a few 
entries of interest from the content point of 
view. They will be compared with the answers of 
the self-respondents and analyzed for their con- 
tent implications. Questions which have shown 
themselves to be particularly difficult and ambi- 
guous are likely to be tested in the field, with 
two or three alternative wordings, early in 1968 
to give some input for the questionnaire content 
in the 1968 test(s). All these are modest endea- 
vours but limitations of time and personnel make 
any wider action inadvisable. 


A more ambitious programme is envisaged for 
the later part of 1968 and for 1969 with long-tem 
implications, but it is unlikely that its results 
would have much impact on the remaining parts of 
the testing programme, except possibly on the last 
dress rehearsal in 1969. 


4, EXPJRIENCE WITH ADDPE ‘S REGISTERS 


As explained earlier tiie commercial address 
registers, such as are available in Canada, are 
not likely to be comparable in comprehensiveness 
with lists available in the United States. How- 
ever, investigations into the possibility of ex- 
istence of city directories and their qualities 
continue. A commercial offer has been received to 
build up a list but at a unit price so low that 
the possibility arises that the firm does not ap- 
preciate the high standards expected from such a 
list. 


A preliminary enquiry directed to over 50 
urban centres with a 1966 population of almost 7 
million (or some 35 per cent of the total popula- 
tion of Canada) revealed that most of them main- 
tain an assessment roli or an electric utility 
billing list, usually both (28). A fifth of the 
lists (counting by population size) is in a state 
not easily transferable into machine-readable form 
and a quarter is "not available to outside users". 
Only half of the managers of the lists are "will- 
ing to supply an extra copy", It is not necessary 
to enquire into the extent of overlap between the 
fifth, the quarter and the half, because it is not 
unreasonable to expect that should the purpose be 
explained more fully, most of the lists would be- 
come available. It remains to be determined ex- 
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perimentally whether these lists can be used for 
the purposes of the 1971 censuses either as the 


core of the address register or only to support 
and strengthen a register produced otherwise. 


Two address registers have been built up in 
the Bureau mostly from administrative sources(20), 
and the evidence available with regard to these 
two address registers is described in the follow- 
ing two paragraphs. Inasmuch as both registers 
have been constructed in the towns of Ontario and 
inasmuch as the municipal assessment rolls are 
important elements in both lists it is not certain 
how far the experiences are valid for assessment 
rolls in other provinces with different legisla- 
tive requirements for these rolls. 


While these rolls appear to promise the 
biggest immediate pay-off, our eyes are not closed 
to the possibility of utilizing some other sources 
as well, such as voters! lists, building permits, 
demolition permits, completion records of the 
Central Mortgage and Housing Corporation, City 
directories, postal lists of deliverable address- 
es (11). 


In connection with address registers, refer- 
ence should be made to their great potential for 
uses other than as a vehicle to mail out census 
forms. At least two important potential applica- 
tions should be mentioned. One relates to the 
automatic assignment of geographic co-ordinates to 
the addresses in the register and through the 
register to the census documents. This capability 
might add new dimensions to our ability to re- 
trieve census data for user-specified areas. 
coding is the topic of another paper presented at 
this session (14). The other important potentiai 
application of address registers is during the 
intercensal period as a sampling frame for current 
surveys. This latter potential application is, 
of course, contingent on our ability to keep the 
register up-to-date at a reasonable cost. 


Geo- 


4.1 Kitchener-Waterloo address register 


eee 


The address register for the two neighbour- 
ing towns of Kitchener-Waterloo was built up in 
1966 from the 1961 Census iists of households, the 
current electricity billing lists and the current 
municipal assessment rolls. The detailed tech- 
nical operations which led to it and the results 
have been reported upon in several memoranda (11, 
19, 24, 25, 26, 27). It was no big discovery that 
the out-of-date 1961 Census lists of households 
contained only 65 per cent of the addresses on the 
joint list (26). 


Of the two other sources the assessment 
rolls are clearly superior (90 per cent as against 
83 per cent) but they still miss proportions too 
high to leave to the postal check to make up. We 
intend to investigate why some addresses on the 
electrical billings and in the census list were 
missed from the assessment rolls. 


The 37,000 addresses in Kitchener-Waterloo 
were checked by the Post Office letter carriers 
against the slots which they have on their sorting 
tables. Although some action or another by the 
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letter carriers was required with regard to 
6,000 addresses (e.g., there were apparently 
2,000 duplicates), the genuine additions were a 
mere 2 per cent. However, because,a third of 
about 1.3 per cent of addresses which were with- 
held from the letter carriers for quality control 
Purposes were not reported as missing during the 
postal check it can be assumed that another 1 per 
cent has not been discovered (27). 


4.2 London address register 


The September 1966 listing of households in 
London already mentioned earlier provided not only 
the main source for the distribution of question- 
naires in September 1967, but served also for the 
purposes of the study of address registers. The 
listing was carried out on principles very close 
to those of the monthly Labour Force Survey (15, 
29). 


Whenever possible the listing was to be 
done from external appearance, and enquiries in- 
side households were not encouraged. The opera- 
tion was quality-controlled in the hard-to-enumer- 
ate central part of the town (12). In such areas 
the over-all "“error'' rate was 12 per cent and 
some observers felt that this high rate could be 
combated only through enquiries inside the house- 
hold.15/ 


In May 1967, the list was postal-checked and 
& per cent of new addresses were gained. Of ad- 
dresses withheld from the Post Office (a sampie of 
1,319) almost a quarter was not reported as miss- 
ing by the Post Office (1). It can, therefore, be 
assumed that there was another 1 per cent of ad- 
dresses not found by the Post Office in this ad- 
vance postal check. 


The second postal check, conducted just be- 
fore the D-day of September 12, 1967, added an- 
other 905 addresses, but missed again some ad- 
dresses judging from the 264 householders who re- 
ported not having received mailed questionnaires 
and who were not on the address register.16/ 


A direct measurement of the completeness of 
the list will be attempted in the Post-Enumera- 
tion Survey. The address register has also been 
studied through a comparison with other lists in 
@ manner similar to the study of the Kitchener- 
Waterloo address register (22, 30). 


5. PRESENT PLANS 


5.1 Further methodological tests 


Address registers being the very corner- 
stone of any mailing operation, investigations 
into their reliability, alternative modes of 
building them up and their costing will continue. 
In some areas mailing is not feasible. In the 
absence of other suitable lists and on account of 
low population density, listing in such areas 
could be so expensive that only simultaneous enu- 
meration could be considered with respect to short 
forms. In these areas we may have some experi- 
ments to drop off the long forms and ask respond- 
ents to mail chem back, 


In areas where mailing is possible, but a 
centralized operation to control editing and 
follow-up is not, a substitute method to carry out 
the editing and other operations from the back of 
the enumerator's car would have to be designed and 
tested. In such areas, the arrangements would 
have to be made through local talent and it would 
not be possible to reap the benefit of organiza- 
tion by our regional office personnel. The popu- 
lation concerned might be as much as 35 per cent 
of the country, if we exclude the 45 per cent in 
the 17 metropolitan areas and the agricultural or 
rural areas with, say 20 per cent where the need 
to take a Census of Agriculture simultaneously 
creates special conditions and requirements, 


There is a host of methodological tests 
which, though important, are not likely to be un- 
dertaken because of shortage of personnel: pub- 
licity (the public relations circumstances of a 
national census cannot be satisfactorily simulated 
for a local test), questionnaire format and design 
(linear, columnar, page-per-person, etc.), influ- 
ence of training (which type is effective with 
what kind of people?). 


5.2 Content of questionnaires and mode of 
controlling its quality 


Alternative wordings for several questions, 
where there is reasonable hope for some tangible 
results, are likely to be tried. A few new ques- 
tions are likely to be tried on an experimental 
basis. These endeavours will be carried out in 
the two official languages of the country. 


The attempts to discover the influence of 
editing instructions (both manuai and mechanical) 
have already been described earlier, as well as 
other investigations leading to the assessment of 
the meaning of census questions and answers (e.g., 
23). Another important question to which no an- 
swer will probably be available in time to formu- 
late the 1971 plans is how to strike the balance 
between the expense of editing and the resulting 
content of the questionnaires. High-quality edit- 
ing triggers off follow-up directly and propor- 
tionately. 


5.3 Census users and their contributions 


The main endeavours in the determination of 
questionnaire content are user-oriented. Same 
goes for the outputs, but these are too important 
topics to be treated at the end of an already too 
long paper. A separate section in the Census 
Division has been recently established to develop 
the understanding of users and our understanding 
of their needs. 


FOOTNOTES 


1/ The first nominal census of Canada was taken 
~ in 1666 and covered the then European popula- 
tion of 3,215 persons enumerated (8, p. 9). 

2/ In the middle of the four earlier intercensal 
a periods such censuses were held only in the 

Prairie Provinces experiencing then high mi- 
gration. 
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At least where the functioning of such a cen- 
tral office is a feasible operation. Where 
due to low density it is not practicable, edit 
decisions and follow-up decistons have to be 
taken in less controlled and less independent 
circumstances. 

In areas where a mail census is not possible, 
questionnaires have to be picked up or actual- 
ly completed through canvassing by enumera- 
tors. In such areas the publicity in nearby 
large cities is likely to be a source of dis- 
quiet to conscientious respondents. 

We did not expect the new method to be a cure 
for all our problems. We were aware that the 
improvements in the 1960 US census were small- 
er than hoped for (32). Then there are always 
unexpected teething troubles, such as over- 
imputations of large numbers (32) on a new 
mechanical device. 

Results. ofthe. 1967 test.will begin to become 
available late in 1967, with most of the re- 
sults becoming available in 1968, too late for 
the planning of the 1968 tests. 

But see footnote 15 for an altermative solu- 
tion. 

Assuming that a field list of households is to 
be the prime source of the address register. 
As explained later, this is by no means cer- 
tain. And, of course, this argument would be 
quite invalid should some kind of list-drop- 
off (at least the long questionnaire) -mail- 
back be adopted. 

Whether the winter months while the building 
activity is at its lowest make really much 
difference to an address register, remains 


“still to be determined. 


Actually in London there was a sixth source: 
publicity. Some 264 respondents telephoned 
that they nad not received questionnaires and 


- were found not to have been on the list (an- 


other 243 were found to be on the list and 
were given duplicate questionnaires). 

In long-standing Canadian census parlance, 
households are identical with dwellings. It 
is hoped with the new concepts relating to ad- 
dress registers they will also be identical 
with subaddresses. A basic address is one 
street address which may have one or several 
subaddresses (apt. 1, apt. 2, etc.) within the 
same basic address. 

Before the description of the Post-Enumeration 
Survey, a qualification should be stated: it 
is the experience both in the United States 
and Canada that such evaluation surveys appear 
to be more successful in uncovering missed 
subaddresses and even more so basic addresses, 
but seem to be less successful in finding per- 
sons missed in partly enumerated households. 
This appears to be a less successful part of 
the main London questionnaires, at least it 
seems to be inferior to the back page of the 
New Haven questionnaire. 

The office coding of families (and the manual 
coding of the mere intricate relations to head 
of household, as well as some other minor man- 
ual entries) delay the processing of the 
"short'' part of the questionnaire. In the pro 
cessing of the 1966 Census, families were cre- 
ated mechanically as they will be in 1971. 

No figures are available on the increases in 
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16/ 


(13) 


(15) 


costs due to such procedure. Enquiries in- 
side households, if uniformally imposed, 
would incidentally provide an opportunity to 
determine whether households should be sent 
an English or a French questionnaire. 

See footnote 10 for other details. 
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